Skip to content

feat(#270): require N consecutive successes in HALF_OPEN before closing circuit#350

Closed
samsolo247 wants to merge 1 commit into
ChainForgee:mainfrom
samsolo247:feat/issue-270-circuit-breaker-half-open-threshold
Closed

feat(#270): require N consecutive successes in HALF_OPEN before closing circuit#350
samsolo247 wants to merge 1 commit into
ChainForgee:mainfrom
samsolo247:feat/issue-270-circuit-breaker-half-open-threshold

Conversation

@samsolo247

Copy link
Copy Markdown
Contributor

Summary

Closes #270

Fixes premature circuit-breaker closure (backlog item #62). Previously record_success() transitioned HALF_OPEN → CLOSED on the very first success — a single lucky probe could reopen a broken provider. This PR adds a configurable success_threshold_in_half_open (default 2) so the breaker only returns to CLOSED after N consecutive successes.

Changes

app/ai-service/services/circuit_breaker.py

What Detail
New constructor param success_threshold_in_half_open: int = 2
New internal counter _half_open_success_count — reset on every OPEN→HALF_OPEN entry and on any failure
record_success() Increments counter in HALF_OPEN; only closes circuit when threshold is reached
record_failure() Resets _half_open_success_count = 0 before reopening — next probe window starts clean
allow_request() Resets _half_open_success_count = 0 on each OPEN→HALF_OPEN transition

All state transitions remain thread-safe under the existing Lock.

app/ai-service/tests/test_circuit_breaker.py

6 new tests in TestHalfOpenSuccessThreshold:

Test Purpose
test_threshold_3_two_successes_remain_half_open AC 1 — 2 successes with threshold=3 → still HALF_OPEN
test_threshold_3_third_success_closes AC 2 — 3rd success → CLOSED
test_default_threshold_is_2 Default behaviour
test_threshold_1_single_success_closes Backward-compatible single-success mode
test_failure_resets_success_count_and_reopens Mid-probe failure wipes accumulated count
test_allow_request_resets_count_on_reentry Each probe window is independent

2 existing tests updated to reflect the new default threshold of 2.

Test run

15 passed in 1.88s

All 15 tests pass (9 pre-existing + 6 new).

…efore closing circuit

Implements issue ChainForgee#270 (backlog item ChainForgee#62).

Problem: record_success() transitioned HALF_OPEN → CLOSED on the very first
success, risking premature closure after a single lucky probe.

Changes
-------
services/circuit_breaker.py
  - New constructor param success_threshold_in_half_open (default 2).
  - New internal counter _half_open_success_count, reset to 0 whenever
    the breaker enters or re-enters HALF_OPEN (allow_request) and on any
    failure (record_failure).
  - record_success() in HALF_OPEN now increments the counter and only
    transitions to CLOSED once the threshold is reached.
  - record_failure() resets _half_open_success_count before reopening,
    ensuring the next probe window starts from zero.
  - All state transitions remain thread-safe (same Lock).
  - Docstrings updated.

tests/test_circuit_breaker.py
  - test_circuit_breaker_basic_transitions updated: default threshold=2
    means first success stays HALF_OPEN, second closes.
  - TestHalfOpenSuccessThreshold (new class, 6 tests):
      • threshold_3_two_successes_remain_half_open  — AC ChainForgee#1
      • threshold_3_third_success_closes            — AC ChainForgee#2
      • default_threshold_is_2
      • threshold_1_single_success_closes (backward-compat)
      • failure_resets_success_count_and_reopens
      • allow_request_resets_count_on_reentry
  - TestCircuitBreakerMetrics.test_success_closes_circuit_and_resets_state_gauge
    updated to issue two record_success() calls (matches default threshold=2).
@samsolo247 samsolo247 closed this Jul 16, 2026
@samsolo247
samsolo247 deleted the feat/issue-270-circuit-breaker-half-open-threshold branch July 16, 2026 09:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make circuit_breaker retry after HALF_OPEN re-tries with backoff

2 participants